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Oz 

Amac: Bu ¢alismada, akromiyoklavikiiler eklem (AKE) yaralanmalarinda kul- 
lanilan Rockwood siniflamasinin ortopedi asistanlarinda gézlemci ici ve goz- 
lemciler arasi giivenilirliginin arastirilmasi amaclandi. Gere¢ ve Yontem: Ak- 
romiyoklavikiiler eklem yaralanmasini gosteren omuz 6n-arka grafilerinin bu- 
lundugu 37 slaytlik bir Microsoft® Power Point (PPT) sunumu hazirlandi. Top- 
lam 15 ortopedi asistanindan PPT sunusundaki yaralanmalari Rockwood si- 
niflamasina gore siniflandirmalari istendi. U¢ ay sonra slaytlarin yerleri de- 
istirilerek degerlendirme tekrar yapildi. Gozlemci ici ve gézlemciler ara- 
si uyumu degerlendirmek icin intraclass correlation coefficient(ICC) degerle- 
ri hesaplandi. Bulgular: Tim arastirmacilar g6z oniine alindiginda, gozlemci 
igi uyum icin ICC degeri 0.732 (0.476-1) , gézlemciler arasi uyum i¢in ICC de- 
Beri 0.690 (0.654-0.739) olarak bulundu. Asistan hekimlerin 6zellikle tip 2-3 
ve tip 3-5 ayirimi yapmakta zorlandiklar! g6rtildui. Tartisma: Calismamizin so- 
nuclari; her ne kadar asistan hekimler icin Rockwood siniflamasinin gozlem- 
ciler igi ve arasi giivenilirliginin kabul edilebilir diizeyde oldugunu gostermek- 
teyse de tip 2 ile tip 3 ve tip 3 ile tip 5 yaralanmalarin ayriminin yapilmasi- 
nin zor oldugu dikkat ¢ekmektedir. 
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Abstract 

Aim: The purpose of this study was to evaluate intra and inter-observer reli- 
ability of the Rockwood classification system, which is used to assess acro- 
mioclavicular (AC) joint injuries, in terms of orthopedics residents’ evaluation. 
Material and Method: A Microsoft® Power Point (PPT) presentation with 37 
slides including shoulder anterior-posterior radiographs displaying acromio- 
clavicular joint injuries was prepared. A total of 15 orthopedic residents 
were asked to classify the injuries according to the Rockwood classification 
system. The order of the slides was changed and the assessment was again 
carried out three months later by the same reviewers. In order to assess 
intra- and inter-observer agreement levels, intra-class correlation coefficient 
(ICC) values were calculated. Results: When considering the results of all the 
participants, the kappa value for intra-observer and kappa value for inter-ob- 
server reliability were found to be 0.732 (0.476-1) and 0.690 (0.654-0.739), 
respectively. Analysis indicated that residents had difficulty in differentiat- 
ing type 2-3 and type 3-5 injuries. Discussion: Although our studies’ finding 
was that there was an acceptable intra- and inter-observer reliability for 
Rockwood classification among orthopedics residents, it was evident that 
differentiating type 2 and type 3 injuries and differentiating type 3 and 5 
injuries was difficult. 


Keywords 
Rockwood Classification; Resident; Inter-Observer; Intra-Observer 


DOI: 10.4328/JCAM.4876 


Received: 30.11.2016 Accepted: 27.12.2016 Printed: 01.07.2017 


J Clin Anal Med 2017;8(4): 315-9 


Corresponding Author: Emre Bilgin, Ortopedi ve Travmatoloji Klinigi, Tepecik Egitim ve Arastirma Hastanesi, Konak, Izmir, Turkey. 


T.: +90 2324696969-1415 E-Mail: dremrebilgin@hotmail.com 


Journal of Clinical and Analytical Medicine | 


515 


INTER- AND INTRA-OBSERVER RELIABILITY OF ROCKWOOD CLASSIFICATION 


Introduction 

Acromioclavicular (AC) joint injuries constitute approximately 
9% of shoulder girdle injuries. AC injuries are frequently seen 
in males and young athletes and most commonly occur as a 
result of falling on the shoulder [1]. The force that occurs as 
a result of falling on the shoulder directs the acromion to the 
medial and inferior. Based on the force intensity, a stress oc- 
curs in the acromioclavicular ligament, then a rupture in the 
coracoclavicular ligaments, and finally separation of the deltoid 
and trapezius muscles can occur. This pathologic characteristic 
of acromioclavicular joint dislocation was first described by Ca- 
denat. Tossy et al. have classified these injuries as types 1, 2, 
and 3. [2]. Because this classification system considers the dis- 
placement only in the frontal plane, it does not include all of the 
possible injuries. Rockwood et al. has expanded the classifica- 
tion by adding types 4, 5, and 6 [3]. The Rockwood classification 
system is being used in both diagnosing AC joint injuries and in 
planning the treatment, whether conservative or surgical. Type 
1 and type 2 injuries are undisplaced injuries and can be con- 
servatively treated, without surgery. The treatment of type 3 
injuries with superior displacement of the clavicle as a result of 
acromioclavicular and coracoclavicular ligament rupture is con- 
troversial [4-8]. Since type 5 injuries show advanced displace- 
ment, a surgical treatment is suggested for such injuries [3, 9]. 
It is difficult to differentiate type 3 and type 5 injuries by the 
evaluation of AP radiographies. Type 3 injuries are character- 
ized as 100% displacement of the coracoclavicular space; a dis- 
placement between 100-300% is classified as a type 5 injury. 
The clavicle is displaced posteriorly into the trapezius muscle in 
type 4 injuries. The diagnosis of these injuries is made easier 
by adding axial radiography or computerized tomography [10]. 
There are a few studies of Rockwood classification’s inter- 
and intra-observer reliability in the literature. In light of these 
studies, the dislocation level of the acromioclavicular joint is 
questionable and may not be useful for surgeons to choose the 
appropriate treatment modality [11-14]. These injuries are gen- 
erally assessed by the residents who specialize in orthopedics in 
the emergency departments. Therefore, we aimed to evaluate 
the intra- and inter-observer reliability of the Rockwood clas- 
sification among orthopedics residents. By considering that the 
reliability of this classification system increases with experi- 
ence, our hypothesis was that the intra- and inter- observer 
reliability of this classification would be low among residents. 


Material and Method 

The patients admitted to the emergency department with the 
diagnosis of AC joint injury between January 2013 and Decem- 
ber 2014 were retrospectively evaluated from the hospital’s 
digital database. The study was approved by the local ethics 
committee. Inclusion criteria were: patients with both coraco- 
clavicular spaces visible on the same radiograph, patients with 
acute injury, patients with unilateral injury, and patients with 
closed clavicular physis. The radiographies of the 51 patients 
with AC joint injury were assessed. 12 patients whose shoulder 
joints were not seen in the same radiography and 2 patients 
with chronic AC joint dislocation were excluded from the study. 
37 patients whose distal clavicular physis had been closed and 
with unilateral injury were included in the study. The anterior- 


posterior shoulder and postero-anterior lung radiographs were 
taken from the hospital’s digital system and were then recorded 
in the Joint Photographic Experts Group (JPEG) format. A Micro- 
soft® Power Point (PPT) presentation was made with 37 slides, 
each of which had a single patient’s radiographs, selected by a 
non-participant of the study. A total of 15 orthopedic residents 
participated in the study. Each resident was asked to examine 
the slide and to classify the injuries with the Rockwood clas- 
sification system. Explanations and figures related to the clas- 
sification system were quoted from a well-known orthopedic 
trauma book and were distributed to the study participants. 
Each participant individually carried out the assessment. After 
three months, the order of the slides was changed. The new 
order of the slides was recorded to assess the intra-observer 
reliability. The participants were then asked to re-evaluate 
the radiographs; they were given two weeks to complete the 
assessment. The second assessment’s results were organized 
for the calculation of intra-observer reliability. Residents were 
evaluated by number of working years and kappa values were 
statistically compared. 

Statistical analysis was performed by calculating the coeffh- 
cient kappa value using SPSS 17 statistical software for in- 
ter- and intra-observer reliability. The inter-observer reliability 
kappa value was calculated by comparing the first classification 
between participants. Intra-observer reliability kappa value was 
calculated by comparing the first and the second classification 
for each observer. The kappa value coefficient was interpreted 
according to the guidelines proposed by Landis and Koch: less 
than 0.00 poor, 0.00-0.20 slight, 0.21-0.40 fair, 0.41-0.60 mod- 
erate, 0.61-0.80 substantial agreement, and 0.81-1.00 almost 
perfect agreement [15-16]. 


Results 

The kappa value for inter-observer reliability was 0.690 (0.654- 
0.739) (substantial agreement). The mean kappa value for intra- 
observer reliability was calculated 0.741 (0.476-1) (substantial 
agreement). Kappa values of intra-observer reliability for each 
observer are shown in Table 1. Intra-observer reliability was 


Table 1. Kappa values of intra-observer reliability for each observer 


Training Year Observer Number Kappa values (min-max) 


2 1 0.672 (0.449-0.817) 
2 2 0.827 (0.689-0.907) 
2 3 | 

2 4 0.476 (0.183-0.692) 
2 5 1 

2 6 0.654 (0.422-0.805) 
3 7 0.776 (0.606-0.878) 
3 8 0.637 (0.398-0.795) 
4 9 0.734 (0.541-0.854) 
4 10 0.512 (0.228-0.712) 
4 11 1 

5 12 0.635 (0.395-0.794) 
5 13 | 

5 14 0.515 (0.233-0.717) 
5 15 0.544 (0.270-0.736) 
Mean 0.741 (0.476-1) 
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highest in second-year residents (kappa value of second-year 
residents: 0.771, fourth year residents: 0.748, third year resi- 
dents: 0.706, fifth year residents: 0.673). Results of the first and 
second reviews according to each Rockwood type are shown in 
Tables 2 and 3. It was observed that participants particularly 
had difficulty differentiating types 2-3 and types 3-5 (figures 
1 and 2). 


Figure 1. Difficulty in differentiating types 3 and 5 (case number 3) 


Figure 2. Difficulty in differentiating types 2 and 3 (case number 21) 


Discussion 

The ideal classification system should be simple, reliable, and 
repeatable and should be used as a guide for treatment and 
prognosis. In addition, such a system should provide a common 
and standard language for both clinicians and researchers. The 
diagnosis and treatment of AC joint injuries are compulsory 
for orthopedic surgeons. The Rockwood classification system 
is most frequently used for these types of injuries. In the litera- 
ture, there are few studies assessing the intra- and inter-ob- 
server reliability of the Rockwood classification system [11-14]. 
Kraeutler et al. have researched this classification’s reliability in 
types 3, 4, and 5 AC joint dislocations and found the intra- and 
inter-observer reliabilities as good [11]. Cho et al. reported the 
intra-observer reliability as moderate, and the inter-observer 


Table 2. Results of the first review according to each Rockwood type 


Rockwood Types 


Patient Number 1 (n) 2 (n) 3 (n) 4 (n) 5 (n) 6 (n) 
1 2 11 1 1 
2 1 13 1 
3 7 1 Zs 
4 4 10 1 
5 1 5 3 2 4 
6 10 5 
7 5 10 
8 3 9 3 
9 12 3 
(0) 6 7 2 
1 1S 2 
2 3 1 11 
3 1 11 5 
4 1 9 5 
5 13 1 1 
6 4 10 1 
7 5 1 9 
8 4 8 2 1 
9 3 >) 5 2 
20 6 6 5 
21 7 8 
22 7 8 
23 7 8 
24 5 5 5 2 
25 2 13 
26 12 3 
27 8 7 
28 3 12 
29 1 14 
30 4 11 
31 2 3 10 
32 12 2 1 
33 7 5 3 
34 5 10 
35 14 1 
36 9 6 
37 3 12 


reliability as fair [14]. Ng et al. assessed the Rockwood, Tossy, 
and Allman classification systems on Zanca radiographs and 
reported the reliability as weak in all systems [13]. In our study, 
although intra- and inter-observer reliability of Rockwood clas- 
sification system was generally found as good, it was noted 
that differentiating types 2 and 3, and types 3 and 5 is ex- 
tremely difficult. 

Differentiating type 3, 4, and 5 injuries is important in terms of 
treatment planning. While the type 5 dislocations are treated 
surgically, there is not a consensus about type 3 injuries, and 
these injuries are usually treated conservatively [4-9]. The dif- 
ferentiation between type 3 and type 5 in Rockwood’s classic 
description has been made by comparing the distance between 
the upper surface of the coracoid and the inferior surface of 
the clavicle [3]. Orthopedic surgeons generally do not follow 
this principle, but more commonly assess this separation by 
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Table 3. Results of the second review according to each Rockwood type 


Rockwood Types 


Patient Number 1 (n) 2 (n) 3 (n) 4 (n) 5 (n) 6 (n) 
1 3 11 1 
2 1 13 1 
3 10 5 
4 4 11 

5 5 4 2 4 
6 1 10 4 
7 5 9 1 

8 2 10 2 1 
9 | 11 3 
10 i) 9 1 
11 10 4 1 
12 3 1 11 

13 3 11 1 

14 7 7 1 
15 14 1 
16 5 8 2 
17 9 1 5 
18 3 8 1 

19 4 10 1 

20 1 7 7 

21 7 8 

22 5 9 1 

23 8 7 

24 5 8 2 
25 3 11 1 
26 1 11 3 

27 9 6 

28 5 9 1 

29 1 1 tS 
30 4 11 

31 2 5 8 

32 12 3 

355 8 3 3 1 
34 7 1 7 
35 1 14 

36 10 5 
37 1 14 


observing the relationship of the clavicle with the acromion in 
the shoulder antero-posterior radiography. We think that such 
assessment may lead to confusion in typology. Both in the study 
by Kraeutler et al. and in our study, a consensus has not been 
built regarding differentiating the classification of type 3 and 
type 5 injuries [11]. This complicates the treatment decision of 
the surgeon. 

Observers first assessed the AC joint injuries by stress radio- 
graphs of the shoulder with the Rockwood classification system 
in the study of Schneider et al. Then, the coracoclavicular dis- 
tances in the shoulder injured and the uninjured shoulder were 
digitally measured, and by comparing the distances, a coraco- 
clavicular (CC) index was obtained. In the assessment using the 
classic Rockwood classification, they found intra- and inter-ob- 
server reliability to be good, but after supplementing this with 
thick index, they found the intra- and inter-observer reliability 


to be perfect. As a result, they have suggested that the mea- 
surement of CC should be added to the Rockwood classification 
in acute AC joint injuries [12]. 

In most studies, it has been shown that the standard anterior- 
posterior shoulder radiography may not be sufficient in the di- 
agnosis of type 4 injuries [7, 8, 10]. It is not always possible to 
take axial radiography due to acute pain. Therefore, other diag- 
nostic methods such as ultrasound, magnetic resonance imag- 
ing, or computerized tomography can be beneficial in diagnosis 
of these type injuries. In the study of Cho et al., 3-dimension 
computerized tomography has also been used in additional to 
conventional radiographs. The authors stated that the addition 
of tomography had not increased the classification system's re- 
liability. Therefore, the authors concluded that the use of com- 
puterized tomography in the diagnosis of the AC joint injuries 
was unnecessary, considering the high radiation dose exposure 
and the cost [14]. However, this is a debatable issue and further 
studies are needed to research the effect of computerized to- 
mography on this classification’s reliability. 

Schaefer et al. have claimed that it is possible to assess both 
coracoclavicular and acromioclavicular ligaments by magnetic 
resonance imaging (MRI). The authors also suggested that MRI 
may be used as a direct diagnostic method in AC joint injuries. 
In addition, the authors have stated that treatment method 
may change when MRI information was added to the radio- 
graphs [17]. 

There are some limitations of our study. The assessments were 
carried out based on shoulder anterior-posterior radiographs. 
Adding axial shoulder radiographs, computerized tomography, 
or MRI could affect the reliability of this classification system. 
Only orthopedic residents were included in the study so the ef- 
fect of experience on the reliability of this classification system 
was not evaluated because more experienced surgeons were 
not included. 

In conclusion, intra- and inter-observer reliability of the Rock- 
wood classification system were found to be good although the 
experience levels of the participants were limited. However, 
there were difficulties in differentiating types 2-3 and 3-5 in- 
juries. This confusion can cause either under-treatment or un- 
necessary over-treatment. In such conditions, additional imag- 
ing modalities such as computerized tomography or MRI can 
be beneficial. 
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